1. Field of Invention
The present invention relates to the field of computer software and computer program testing. More specifically, it relates to verification of graphics images produced by a computer graphics program.
2. Discussion of Related Art
In the computer software industry, the amount of time needed to test software products, specifically computer graphics programs, has become a significant bottleneck in the product development timeline. The testing cycle for graphics programs, where images typically must undergo thousands of tests, presently takes several weeks. The test cycle is typically conducted after the software product has been completed by the developer and must be completed before the product is released.
With a typical computer graphics program, the test engineer must verify that the software produces output images, as expected, with no significant errors. The images generated by the computer program must be within an acceptable range of images described in the program's specification. Regardless of how confident application developers are with the integrity and performance of the software, before the software is commercially released, test data is needed to support the assertion that the product runs as expected. The bottleneck occurs in creating this test data. by the increasing complexity of graphics libraries, which contain routines to generate images and is required when running a graphics program.
Efforts to reduce the number of reference images have led to other drawbacks which can be explained by the fact that computer graphics use mathematical models to simulate scence designed by the user. Whenever mathematical approximations are used, there are going to be errors involved. With image comparison verification, because one of the goals is to reduce the number of reference images, test engineers tend to combine many test programs into one test frame. This increases the error margin in the test engineer's prediction of the expected image, since a higher number of mathematical models is used in one test frame. Consequently, the test engineer usually describes the expected output image using general terms, such as "a shaded red square with white shading in the lower right corner." The problem with this type of description is that different people looking at the same image may focus on different areas and may focus on portions different from the portion focused on by the test engineer designing the test case.
Another time-consuming factor arises from a pixel being in error due to more than one bug in the graphics program. Cumulative errors may cause a pixel in an output image to be different from a pixel in a reference image. A senior engineer typically has to make a judgment call as to the significance of that pixel. Because some errors are more significant than others, it must be determined which errors can be ignored and which ones need to be addressed. This determination typically requires the experience of a senior test engineer, who also cannot always determine which errors are significant and which ones are not.
FIGS. 1a-c show the different images needed for image comparison verification in the prior art. FIG. 1a shows a screen shot of a typical reference image of various geometric shapes and lines. FIG. 1b is a screen shot of a typical corresponding output image which is to be tested against the reference image of FIG. 1a. FIG. 1c is a screen shot of an image showing the differences, pixel by pixel, between the output and reference images.
As is evident from the discussion above, there are several significant drawbacks from using traditional image comparison testing. First, each hardware/software configuration requires a separate set of reference images resulting in a proliferation of images. Second, the testing method does not provide qualitative information on failed cases. There is a need to generate a new set of reference images whenever there is a new hardware/software configuration for the program. Third, since there are many possibilities or reasons a test may fail it is necessary to have a senior test engineer supervise the error checking step. Finally, there is no room for error tolerance which would eliminate insignificant or "don't care" type rendering differences between the images.
Another method of testing computer graphics programs is referred to as sample point testing. With this type of testing, pixel values are hardcoded in the test programs and are given only a color tolerance. The test data determines what type of testing is going to be applied to that pixel and the area around that pixel (the location of the center of the area is hardcoded in the test program). However, the hardcoded values of the pixels apply only to a particular hardware/software configuration.
A major problem arises in that maintaining and updating the test programs is a very time consuming and error prone procedure. Essentially, the test program code has to be changed every time there is any type of update. Having the hard coded expected values of selected points contained within the test program itself makes the test platform dependent. Thus, the significant shortcomings of sample point verification are that it provides limited error checking and that the expected values of the points are bard coded into the test program. Even error tolerances which may be included in the test program cannot be modified for individual test cases without having to modify the test program itself.
Therefore, what is needed is a method of testing computer graphics programs that is efficient and eliminates insignificant rendering differences. It would also be desirable to have a testing methodology in which the errors can be identified and qualified with a high degree of confidence and, at the same time, not require a high level of maintenance of reference images. It is also desirable to have more modular type testing so that errors can be identified and corrected in less time.